Systems and Methods for Object Manipulation with Haptic Feedback

ABSTRACT

One illustrative computing device disclosed herein includes a sensor configured to detect a user interaction with a physical object and transmit a sensor signal associated with the user interaction. The illustrative computing device also includes a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a characteristic of the physical object based on the sensor signal; determine a haptic effect associated with the characteristic; and transmit a haptic signal associated with the haptic effect. The illustrative computing device further includes a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to U.S. patent application Ser. No. ______,filed the same day as the present application and entitled “Systems andMethods for Haptically-Enabled Interactions with Objects,” (AttorneyDocket No. IMM548 (58518-922294)), the entirety of which is herebyincorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to the field of user interface devices.More specifically, the present invention relates to object manipulationwith haptic feedback.

BACKGROUND

Humans are increasingly using computer-based systems for a variety ofevery-day activities. For example, consumers may use mobile devices(e.g., smartphones) to read reviews about a product or compare prices ofa product at different stores while shopping. It may be difficult andtime consuming, however, for a consumer to interact with the small userinterface (e.g., touchscreen display) of a mobile device to obtainrelevant information while shopping. Thus, there is a need for animproved user interface that can provide such information to usersquickly and easily. It may be desirable to use haptic feedback (e.g.,mechanical vibrations) to improve such user interfaces.

SUMMARY

Embodiments of the present disclosure comprise object manipulation withhaptic feedback. In one embodiment, a computing device of the presentdisclosure may comprise: a sensor configured to detect a userinteraction with a physical object and transmit a sensor signalassociated with the user interaction. The computing device may alsocomprise a processor in communication with the sensor, the processorconfigured to: receive the sensor signal; determine a characteristic ofthe physical object based on the sensor signal; determine a hapticeffect associated with the characteristic; and transmit a haptic signalassociated with the haptic effect. The computing device may furthercomprise a haptic output device in communication with the processor, thehaptic output device configured to receive the haptic signal and outputthe haptic effect.

In another embodiment, a method of the present disclosure may comprise:receiving a sensor signal from a sensor, wherein the sensor signal isassociated with a user interaction with a physical object. The methodmay also comprise determining a characteristic of the physical objectbased on the sensor signal; and determining a haptic effect associatedwith the characteristic. The method may further comprise transmitting ahaptic signal associated with the haptic effect to a haptic outputdevice, wherein the haptic output device is configured to receive thehaptic signal and output the haptic effect. Yet another embodimentcomprises a computer-readable medium for implementing such a method.

These illustrative embodiments are mentioned not to limit or define thelimits of the present subject matter, but to provide examples to aidunderstanding thereof. Further embodiments are discussed in the DetailedDescription, and additional description is provided there. Advantagesoffered by various embodiments may be further understood by examiningthis specification and/or by practicing one or more embodiments of theclaimed subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

A full and enabling disclosure is set forth more particularly in theremainder of the specification. The specification makes reference to thefollowing appended figures.

FIG. 1 is a block diagram showing a system for object manipulation withhaptic feedback according to one embodiment;

FIG. 2 is another block diagram showing a system for object manipulationwith haptic feedback according to another embodiment;

FIG. 3 shows an embodiment of a system for object manipulation withhaptic feedback;

FIG. 4 shows another embodiment of a system for object manipulation withhaptic feedback;

FIG. 5A shows still another embodiment of a system for objectmanipulation with haptic feedback;

FIG. 5B shows yet another embodiment of a system for object manipulationwith haptic feedback;

FIG. 6 shows another embodiment of a system for object manipulation withhaptic feedback;

FIG. 7 shows still another embodiment of a system for objectmanipulation with haptic feedback;

FIG. 8 shows yet another embodiment of a system for object manipulationwith haptic feedback; and

FIG. 9 is a flow chart of steps for performing a method for providingobject manipulation with haptic feedback according to anotherembodiment.

DETAILED DESCRIPTION

Reference will now be made in detail to various and alternativeillustrative embodiments and to the accompanying drawings. Each exampleis provided by way of explanation, and not as a limitation. It will beapparent to those skilled in the art that modifications and variationscan be made. For instance, features illustrated or described as part ofone embodiment may be used in another embodiment to yield a stillfurther embodiment. Thus, it is intended that this disclosure includemodifications and variations that come within the scope of the appendedclaims and their equivalents.

Illustrative Examples of Object Manipulation with Haptic Feedback

One illustrative embodiment of the present disclosure comprises acomputing device. The computing device comprises a processor coupled toa memory via a bus. In the illustrative embodiment, the computing deviceis configured to be worn by the user. For example, the computing devicemay comprise a ring configured to be worn on a user's finger.

In the illustrative embodiment, the computing device comprises a sensorconfigured to detect a user interaction with an object and transmitsensor signals to the processor. An object, as used herein, is anything(e.g., real or virtual) with which a user can potentially interact. Forexample, while browsing a grocery store, the user may touch a milkbottle. The computing device may detect, via the sensor, that the usertouched the milk bottle. In the illustrative embodiment, the computingdevice determines a characteristic associated with the object based onthe sensor signals. For example, the computing device may determine thebrand of the milk and communicate with one or more servers (e.g., viathe Internet) to determine the ingredients of the milk. Further, in theillustrative embodiment, the computing device determines if thecharacteristic associated with the object matches a criterion. Forexample, the user may be allergic to soy. The user may have input thisallergen information into the computing device. The computing device mayanalyze the ingredients of the milk to determine if the milk containssoy. In the illustrative embodiment, if the characteristic matches thecriteria, the computing device outputs a haptic effect via a hapticoutput device. For example, if the contents of the milk include soy, thecomputing device may output a haptic effect, e.g., a shock sensation.This may help the user make healthy purchasing decisions.

As another example, a user may be looking for a good deal at a discountclothing store. The discount clothing store may have a box of discountclothing that is all mixed together. The user may input her shirt sizeinto the computing device. In the illustrative embodiment, as the usermoves a hand through the box of mixed clothing, the computing devicedetermines the barcode of (or a RFID code for) each piece of clothingthat the user contacts. The computing device further communicates with aserver (e.g., associated with the store) to determine the size of eachpiece of clothing based on the barcode (or RFID code). In theillustrative embodiment, if the size of a piece of clothing matches theuser's shirt size, the computing device may output a haptic effect,e.g., a short vibration. This may help the user sort through a largequantity of products quickly and easily.

Another illustrative embodiment comprises a computing device configuredto detect a user interaction with an object. The computing device thendetermines information such as the brand, manufacturer, distributer,and/or producer associated with the object. Further, in the illustrativeembodiment, the computing device outputs a haptic effect associated withthis information via a haptic output device.

For instance, a manufacturer may associate a unique haptic effect orseries of haptic effects with the manufacturer's company. These uniquehaptic effects can be referred to as a “haptic brand.” For example,Company A may have a haptic brand comprising a series of vibrationscorresponding to the word “fun” in code, e.g., Morse code or anothercode. As another example, Company A may have a haptic brand comprisingthree vibration pulses, with each successive pulse having a largeramplitude than the previous pulse. Such a haptic brand may convey funand excitement to a user. As still another example, Company A may have ahaptic brand associated with audio or visual branding. In one suchembodiment, Company A may have a haptic brand associated with thecompany's audio jingle. For instance, the haptic brand may includevibrations with magnitudes and/or frequencies associated with the notesin the audio jingle. The manufacturer may determine the haptic brandbased on the target market of the manufacturer, the target market of aparticular product, user perception of a haptic effect (e.g., whetherthe haptic effect is perceived as playful, serious, painful, orsatisfying), and/or other criteria.

In the illustrative embodiment, the computing device outputs a hapticbrand based on a user interaction with an object. For example, a usermay be shopping for a new mobile device. The user may pick up a box fora smartphone that is manufactured by Company A. In the illustrativeembodiment, the computing device determines that the user picked up anobject manufactured by Company A. The computing device may determine(e.g., by consulting a lookup table or querying a server via theInternet) a haptic effect comprising vibrations corresponding to theword “fun” in Morse code. In the illustrative embodiment, the computingdevice then outputs the haptic effect. This may allow a user todetermine that the manufacturer of the smartphone is Company A, even ifthe user is not visually focused on the product. The haptic brand may bean effective marketing tool, because it can allow a user to identify theproduct, even if the user is not visually focused on the product. It canalso associate a unique physical interaction with a brand, which canhelp distinguish the brand and provide a more immersive marketingexperience to the user.

The description of the illustrative embodiment above is provided merelyas an example. Various other embodiments of the present invention aredescribed herein and variations of such embodiments would be understoodby one of skill in the art. Advantages offered by various embodimentsmay be further understood by examining this specification and/or bypracticing one or more embodiments of the claimed subject matter.

Illustrative Systems for Object Manipulation with Haptic Feedback

FIG. 1 is a block diagram showing a computing device 101 for objectmanipulation with haptic feedback according to one embodiment. In someembodiments, the computing device 101 may comprise a graspable device(e.g., a smartphone, tablet, e-reader, and/or a portable gaming device).In other embodiments, the computing device 101 may comprise a wearabledevice (e.g., a ring, a shoe, an armband, a sleeve, a jacket, glasses, aglove, a watch, a wristband, a bracelet, an article of clothing, a hat,a headband, and/or jewelry).

In some embodiments, the components (e.g., the processor 202, network110, interaction sensor 132, sensor 130, etc.) of the computing device101 may be integrated into a single housing. In other embodiments, thecomponents may be distributed (e.g., among multiple housings orlocations) and in electrical communication with one another. Thecomputing device 101 may or may not comprise all of the componentsdepicted in FIG. 1. For example, in some embodiments, the computingdevice 101 may comprise the processor 102, bus 106, memory 104, network110, and haptic output device 118.

The computing device 101 comprises a processor 102 interfaced with otherhardware via bus 106. A memory 104, which can comprise any suitabletangible (and non-transitory) computer-readable medium such as RAM, ROM,EEPROM, or the like, may embody program components that configureoperation of the computing device 101. In some embodiments, thecomputing device 101 may further comprise one or more network interfacedevices 110, input/output (I/O) interface components 112, and additionalstorage 114.

Network interface device 110 can represent one or more of any componentsthat facilitate a network connection or otherwise facilitatecommunication between electronic devices. Examples include, but are notlimited to, wired interfaces such as Ethernet, USB, IEEE 1394, and/orwireless interfaces such as IEEE 802.11, Bluetooth, near-fieldcommunication (NFC) interfaces, RFID interfaces, or radio interfaces foraccessing cellular telephone networks (e.g., transceiver/antenna foraccessing a CDMA, GSM, UMTS, or other mobile communications network).

I/O components 112 may be used to facilitate connection to devices suchas one or more displays, touch sensitive surfaces 116, keyboards, mice,speakers, microphones, buttons, and/or other hardware used to input dataor output data. Storage 114 represents nonvolatile storage such asread-only memory, flash memory, ferroelectric RAM (F-RAM), magnetic,optical, or other storage media included in the computing device 101 orcoupled to processor 102.

The computing device 101 may comprise a touch sensitive surface 116.Touch sensitive surface 116 represents any surface that is configured tosense tactile input of a user. One or more touch sensors 108 areconfigured to detect a touch in a touch area (e.g., when an objectcontacts a touch sensitive surface 116) and transmit signals associatedwith the touch to processor 102. Any suitable number, type, orarrangement of touch sensors 108 can be used. For example, resistiveand/or capacitive sensors may be embedded in touch sensitive surface 116and used to determine the location of a touch and other information,such as pressure, speed, and/or direction. As another example, opticalsensors with a view of the touch sensitive surface 116 may be used todetermine the touch position. As still another example, the touchsensitive surface 116 may comprise a LED (Light Emitting Diode) fingerdetector mounted on the side of a display. In some embodiments, touchsensor 108 may be configured to detect multiple aspects of the userinteraction. For example, touch sensor 108 may detect both the speed andpressure of a user interaction, and incorporate this information intothe signal transmitted to the processor 102.

In some embodiments, the computing device 101 comprises a touch-enableddisplay that combines a touch sensitive surface 116 and a display of thedevice. The touch sensitive surface 116 may correspond to the displayexterior or one or more layers of material above components of thedisplay. In other embodiments, touch sensitive surface 116 may notcomprise (or otherwise correspond to) a display, depending on theparticular configuration of the computing device 101.

In some embodiments, the computing device 101 comprises an interactionsensor 132. The interaction sensor 132 is configured to detect aninteraction with the computing device 101 and/or an object (e.g., aproduct on a shelf in a store) by a user (e.g., using a finger, foot,hand, arm, head, leg, or other body part). In some embodiments, the userinteraction may comprise touching the object, gesturing in real space,gesturing using the object (e.g., picking up the object and moving it inreal space), and/or gesturing on an object (e.g., swiping a finger alonga surface of the object). The interaction sensor 132 is furtherconfigured to transmit a sensor signal associated with the interactionto processor 102. The interaction sensor 132 may comprise anaccelerometer, gyroscope, camera, radio frequency identification (RFID)tag or reader, indoor proximity system, NFC communication device, globalpositioning system (GPS) device, magnetometer, ultrasonic transducer,wireless interface (e.g., an IEEE 802.11 or Bluetooth interface),infrared sensor, depth sensor, and/or range sensor.

For example, in one embodiment, the interaction sensor 132 comprises awireless interface that is configured to detect the strength of awireless signal emitted by an object. The interaction sensor 132 maytransmit a sensor signal associated with the wireless signal strength tothe processor 102. Based on the wireless signal strength, the processor102 may determine, for example, whether the computing device 101 iswithin a predefined distance of the object. If so, the processor 102 maydetermine an interaction (e.g., coming within a predefined distance ofthe object) occurred.

In another embodiment, the interaction sensor 132 comprises a cameraoriented toward an object. A user may make a gesture (e.g., a check marksign) in the air (e.g., with a body part, such as a finger, hand, arm,foot, head, or leg) near the object. The interaction sensor 132 maycapture images associated with the gesture and transmit sensor signalsto the processor 102. Based on the sensor signals, the processor 102 maydetermine that a user interaction occurred. The processor 102 mayfurther analyze the sensor signals to determine the specific type ofgesture that occurred. For example, the processor 102 may analyze thesensor signals and determine that the user made a check mark in the airwith a finger.

In some embodiments, the interaction sensor 132 is external to computingdevice 101 and in wired or wireless communication with the computingdevice 101. For example, the interaction sensor 132 may comprise acamera associated with a wearable device (e.g., glasses or a tie) and incommunication with the computing device 101. As another example, theinteraction sensor 132 may comprise a 3D imaging system (e.g., the 3Dimaging system commonly sold under the trademark Microsoft Kinect®) or aLED-based tracking system positioned external to the computing device101 (e.g., on a shelf in a store) and in communication with thecomputing device 101.

The computing device 101 may further comprise one or more additionalsensor(s) 130. The sensor(s) 130 are configured to transmit sensorsignals to the processor 102. In some embodiments, the sensor 130 maycomprise, for example, a humidity sensor, ambient light sensor,gyroscope, GPS unit, accelerometer, range sensor, depth sensor,biosensor, camera, or temperature sensor. In some embodiments, thesensor 130 may be external to computing device 101 and in wired orwireless communication with the computing device 101. For example, thesensor 130 may comprise a biosensor coupled to a wearable device (e.g.,a ring or wristband). The biosensor may be configured to wirelesslytransmit sensor signals to the computing device 101, which may be, forexample, positioned in the user's pocket.

In some embodiments, the computing device 101 comprises a haptic outputdevice 118 in communication with processor 102. The haptic output device118 is configured to output a haptic effect in response to a hapticsignal. In some embodiments, the haptic output device 118 is configuredto output a haptic effect comprising a vibration, a change in aperceived coefficient of friction, a simulated texture, a change intemperature, a stroking sensation, an electro-tactile effect, or asurface deformation (e.g., a deformation of a surface associated withthe computing device 101). Further, some haptic effects may use multiplehaptic output devices 118 of the same or different types in sequenceand/or in concert. Although a single haptic output device 118 is shownin FIG. 1, embodiments may use multiple haptic output devices 118 of thesame or different type to produce haptic effects.

In some embodiments, the haptic output device 118 is external tocomputing device 101 and in communication with the computing device 101(e.g., via wired interfaces such as Ethernet, USB, IEEE 1394, and/orwireless interfaces such as IEEE 802.11, Bluetooth, or radiointerfaces). For example, the haptic output device 118 may associatedwith (e.g., coupled to) a wearable device and configured to receivehaptic signals from the processor 102.

In some embodiments, the haptic output device 118 is configured tooutput a haptic effect comprising a vibration. The haptic output device118 may comprise, for example, one or more of a piezoelectric actuator,an electric motor, an electro-magnetic actuator, a voice coil, a shapememory alloy, an electro-active polymer, a solenoid, an eccentricrotating mass motor (ERM), or a linear resonant actuator (LRA).

In some embodiments, the haptic output device 118 is configured tooutput a haptic effect modulating the perceived coefficient of frictionof a surface associated with the haptic output device 118. In oneembodiment, the haptic output device 118 comprises an ultrasonicactuator. An ultrasonic actuator may vibrate at an ultrasonic frequency,for example 20 kHz, increasing or reducing the perceived coefficient ofan associated surface. In some embodiments, the ultrasonic actuator maycomprise a piezo-electric material.

In some embodiments, the haptic output device 118 uses electrostaticattraction, for example by use of an electrostatic actuator, to output ahaptic effect. The haptic effect may comprise a simulated texture, asimulated vibration, a stroking sensation, or a perceived change in acoefficient of friction on a surface associated with computing device101. In some embodiments, the electrostatic actuator may comprise aconducting layer and an insulating layer. The conducting layer may beany semiconductor or other conductive material, such as copper,aluminum, gold, or silver. The insulating layer may be glass, plastic,polymer, or any other insulating material. Furthermore, the processor102 may operate the electrostatic actuator by applying an electricsignal, for example an AC signal, to the conducting layer. In someembodiments, a high-voltage amplifier may generate the AC signal. Theelectric signal may generate a capacitive coupling between theconducting layer and an object (e.g., a user's finger, head, foot, arm,shoulder, leg, or other body part, or a stylus) near or touching thehaptic output device 118. Varying the levels of attraction between theobject and the conducting layer can vary the haptic effect perceived bya user.

In some embodiments, the haptic output device 118 comprises adeformation device configured to output a deformation haptic effect. Thedeformation haptic effect may comprise raising or lowering portions of asurface associated with the computing device 101. For example, if thecomputing device 101 is positioned within a product (e.g., in a store),the deformation haptic effect may comprise raising portions of a surfaceof a product's packaging to generate a bumpy texture. In someembodiments, the deformation haptic effect may comprise bending,folding, rolling, twisting, squeezing, flexing, changing the shape of,or otherwise deforming a surface associated with the computing device101. For example, the deformation haptic effect may apply a force on thecomputing device 101 or a surface associated with the computing device101, causing it to bend, fold, roll, twist, squeeze, flex, change shape,or otherwise deform. For instance, if the computing device 101 ispositioned within a product's packaging, the deformation haptic effectmay comprise bending the top of a product's packaging toward a user.This may notify the user that the product has characteristic that theuser desires (e.g., that the product is on sale).

In some embodiments, the haptic output device 118 comprises fluidconfigured for outputting a deformation haptic effect (e.g., for bendingor deforming the computing device 101 or a surface associated with thecomputing device 101). For example, the fluid may comprise a smart gel.A smart gel comprises a fluid with mechanical or structural propertiesthat change in response to a stimulus or stimuli (e.g., an electricfield, a magnetic field, temperature, ultraviolet light, shaking, or apH variation). For instance, in response to a stimulus, a smart gel maychange in stiffness, volume, transparency, and/or color. In someembodiments, stiffness may comprise the resistance of a surfaceassociated with the computing device 101 against deformation. In someembodiments, one or more wires may be embedded in or coupled to thesmart gel. As current runs through the wires, heat is emitted, causingthe smart gel to expand or contract. This may cause the computing device101 or a surface associated with the computing device 101 to deform.

As another example, the fluid may comprise a rheological (e.g., amagneto-rheological or electro-rheological) fluid. A rheological fluidcomprises metal particles (e.g., iron particles) suspended in a fluid(e.g., oil or water). In response to an electric or magnetic field, theorder of the molecules in the fluid may realign, changing the overalldamping and/or viscosity of the fluid. This may cause the computingdevice 101 or a surface associated with the computing device 101 todeform.

In some embodiments, the haptic output device 118 comprises a mechanicaldeformation device. For example, in some embodiments, the haptic outputdevice 118 may comprise an actuator coupled to an arm that rotates adeformation component. The deformation component may comprise, forexample, an oval, starburst, or corrugated shape. The deformationcomponent may be configured to move a surface associated with thecomputing device 101 at some rotation angles but not others. Theactuator may comprise a piezo-electric actuator, rotating/linearactuator, solenoid, an electroactive polymer actuator, macro fibercomposite (MFC) actuator, shape memory alloy (SMA) actuator, and/orother actuator. As the actuator rotates the deformation component, thedeformation component may move the surface, causing it to deform. Insuch an embodiment, the deformation component may begin in a position inwhich the surface is flat. In response to receiving a signal fromprocessor 102, the actuator may rotate the deformation component.Rotating the deformation component may cause one or more portions of thesurface to raise or lower. The deformation component may, in someembodiments, remain in this rotated state until the processor 102signals the actuator to rotate the deformation component back to itsoriginal position.

Further, other techniques or methods can be used to deform a surfaceassociated with the computing device 101. For example, the haptic outputdevice 118 may comprise a flexible surface layer configured to deformits surface or vary its texture based upon contact from a surfacereconfigurable haptic substrate (including, but not limited to, e.g.,fibers, nanotubes, electroactive polymers, piezoelectric elements, orshape memory alloys). In some embodiments, the haptic output device 118is deformed, for example, with a deforming mechanism (e.g., a motorcoupled to wires), air or fluid pockets, local deformation of materials,resonant mechanical elements, piezoelectric materials,micro-electromechanical systems (“MEMS”) elements or pumps, thermalfluid pockets, variable porosity membranes, or laminar flow modulation.

In some embodiments, the haptic output device 118 is configured toremotely project haptic effects to a user. For example, the hapticoutput device 118 may comprise one or more jets configured to emitmaterials (e.g., solids, liquids, gasses, or plasmas) toward the user(e.g., toward the back of the user's hand). In one such embodiment, thehaptic output device 118 comprises a gas jet configured to emit puffs orstreams of oxygen, nitrogen, carbon dioxide, or carbon monoxide withvarying characteristics upon receipt of the haptic signal. As anotherexample, the haptic output device 118 may comprise one or moreultrasonic transducers or speakers configured to project pressure wavesin the direction of the user. In one such embodiment, upon the userinteracting with an object, the processor 102 may cause the hapticoutput device 118 to emit a concentrated pressure wave toward the user.The concentrated pressure wave may vibrate a portion of the user's body(e.g., the user's hand).

In some embodiments, the haptic output device 118 may be a portion ofthe housing of the computing device 101. In other embodiments, thehaptic output device 118 may be housed inside a flexible housingoverlaying a surface associated with the computing device 101 (e.g., thefront or back of the computing device 101). For example, the computingdevice 101 may comprise a watch. The haptic output device 118 maycomprise a layer of smart gel overlaying the interior of the band of thewatch. Upon actuating the haptic output device 118 (e.g., with anelectric current or an electric field), the smart gel may expand. Thismay cause the user to perceive a haptic effect comprising a squeezingsensation around the user's wrist.

Turning to memory 104, modules 124, 125, 126, and 128 are depicted toshow how a device can be configured in some embodiments to provideobject manipulation with haptic effects. In this example, the objectinteraction detection module 124 comprises code that configures theprocessor 102 to monitor the interaction sensor 132 and/or theadditional sensors 130 to determine if a user has interacted with anobject. The object interaction detection module 124 may comprise one ormore algorithms or lookup tables useable by the processor 102 todetermine whether a user is interacting with an object.

For example, the computing device 101 may be positioned within a productin a store. A user may, for example, lift the product off a shelf. Theobject interaction detection module 124 may comprise code that samplesthe sensor 130 (e.g., an accelerometer) to track the acceleration of theproduct. If the amount of acceleration exceeds a threshold, the objectinteraction detection module 124 may determine that the product is beingmanipulated (e.g., that the user has lifted the product).

As another example, the interaction sensor 132 may comprise a rangesensor oriented toward an object. The object interaction detectionmodule 124 may comprise code that configures the processor 102 toreceive data from the range sensor. The object interaction detectionmodule 124 may further comprise code that analyzes the data to determinewhether the user is within a certain distance of the object, which maybe indicative of a user interaction.

As still another example, the interaction sensor 132 may comprise a 3Dimaging system oriented toward an object. The object interactiondetection module 124 may comprise code for analyzing images from a 3Dimaging system to determine whether a user is interacting with anobject. Further examples of methods for detecting user interactions withobjects are described with respect to FIG. 9.

In some embodiments, the object may comprise a virtual object. Thevirtual object may be, for example, output on a touchscreen displaycomprising touch sensitive surface 116. The object interaction detectionmodule 124 may comprise code that configures the processor 102 to detecta user interaction with the virtual object. For instance, a user may tapon a location on the touch sensitive surface 116 associated with thevirtual object. The object interaction detection module 124 may receiveone or more sensor signals associated with the user interaction from thetouch sensor 108. In some embodiments, the sensor signals mayincorporate the location, pressure, direction, and/or speed of the userinteraction. The object interaction detection module 124 may determineone or more characteristics of the user interaction based on the sensorsignal.

As another example, the virtual object may be part of an augmentedreality environment output via, for instance, a touchscreen display,goggles, glasses, or contact lenses. The augmented reality environmentmay comprise camera data that has been supplemented (“augmented”) withvirtual content, such as text or images. For example, the interactionsensor 132 may comprise a camera. In some embodiments, the interactionsensor 132 may capture images of the user's dining room, which theprocessor 102 may use to generate the augmented reality environment. Theprocessor 102 may further generate one or more virtual objects withinthe augmented reality environment. For example, in one embodiment, theprocessor 102 may include a virtual table in the augmented realityenvironment. A user may interact with the virtual object, for example,by touching or gesturing in an area in real space associated with thevirtual object. In some embodiments, the object interaction detectionmodule 124 may determine one or more characteristics of the userinteraction, for example, by analyzing images from the interactionsensor 132.

In some embodiments, the object interaction detection module 124comprises code that determines a characteristic (e.g., an amount ofpressure, speed, direction, location, or a gesture) associated with theinteraction. For example, the object interaction detection module 124may comprise code that analyzes sensor signals from the sensor 130and/or interaction sensor 132 to determine the characteristic. In oneembodiment, for example, the object interaction detection module 124 maycomprise code for analyzing images from a 3D imaging system to determinea type of gesture (e.g., swipe, two-finger pinch, shake, etc.) made bythe user.

Object characteristic determination module 125 represents a programcomponent that analyzes data regarding an object to determine acharacteristic associated with the object. The object characteristicdetermination module 125 may comprise one or more algorithms or lookuptables useable by the processor 102 to determine a characteristicassociated with an object.

In some embodiments, the object characteristic determination module 125comprises code that analyzes data from the interaction sensor 132 and/oradditional sensors 130 to determine the characteristic associated withthe object. For example, object characteristic determination module 125may comprise code that receives images of the object from a camera andapplies Optical Character Recognition (OCR) to determine the contents oftext associated with the object. As another example, the objectcharacteristic determination module 125 may comprise code that analyzesimages from the interaction sensor 132 to determine a height, width,color, size, shape, pattern, texture, name, QR code, barcode, label,logo, color scheme, shape, and/or another characteristic of the object.

In some embodiments, the object characteristic determination module 125comprises code that analyzes data received via the network interfacedevice 110 to determine a characteristic of the object. For example, theobject may transmit a signal to the computing device 101 usingBluetooth, IEEE 802.11, RFID, or NFC. The signal may comprise acharacteristic of the object (e.g., the signal may comprise a type,brand, barcode, price, RFID code, or electrical characteristic). Theobject characteristic determination module 125 may analyze parameters ofthe signal to determine the characteristic of the object.

In some embodiments, the object characteristic determination module 125comprises code that determines a characteristic associated with theobject using the Internet or another network (e.g., a LAN). For example,the object characteristic determination module 125 may comprise code forretrieving the name of a sports team (e.g., a baseball team) associatedwith an object (e.g., a baseball bat, helmet, or jersey) bycommunicating with one or more servers or webpages via the Internet. Asanother example, the object characteristic determination module 125 maycomprise code for retrieving nutritional information associated with afood product in a store by communicating with a server associated withthe store (e.g., the store's local server).

In some embodiments, the object characteristic determination module 125comprises code that determines the characteristics of a virtual object.For example, the object characteristic determination module 125 mayconsult with locations in memory 104 to determine characteristics of thevirtual object.

Haptic effect determination module 126 represents a program componentthat analyzes data to determine a haptic effect to generate. The hapticeffect determination module 126 may comprise code that selects one ormore haptic effects to output using one or more algorithms or lookuptables.

In some embodiments, haptic effect determination module 126 comprisescode that determines a haptic effect to output based on a characteristicof the object. For example, the haptic effect determination module 126may determine a haptic effect comprising a number of pulsed vibrationscorresponding to the star rating of a product. As another example, thehaptic effect determination module 126 may comprise code for determininga haptic effect associated with a brand or manufacturer of the object.For instance, Immersion Corporation may associate its brand with ahaptic effect comprising four short pulsed vibrations. Upon the userinteracting with an object produced by Immersion Corporation, the hapticeffect determination module 126 may determine a haptic effect comprisingfour short pulsed vibrations. This may notify the user that the user isinteracting with a product from Immersion Corporation.

In some embodiments, the haptic effect determination module 126comprises code that retrieves a haptic effect and/or a characteristic ofa haptic effect from a server over a network. For example, a user mayinteract with an object produced by Immersion Corporation. The hapticeffect determination module 126 may query Immersion Corporation'sservers (via a network such as the Internet) and receive haptic effectdata (e.g., amplitude, waveform, and/or frequency and timing data). Thehaptic effect determination module 126 may determine a haptic effectbased on the received haptic effect data.

In some embodiments, the haptic effect determination module 126comprises code that determines a haptic effect based on whether acharacteristic of the object meets one or more criteria. For example,the criteria may comprise a list of approved manufacturers for anelectronics product. Upon the user interacting with an electronicdevice, the processor 102 may determine if the electronic device wasproduced by an approved manufacturer. If not, the haptic effectdetermination module 126 may determine a haptic effect comprising abuzzing sensation. As another example, the criteria may comprisedimensions (e.g., a length, width, and height for a piece of furniture).While the user is browsing a furniture store for living room furniture,the user may interact with a couch. The haptic effect determinationmodule 126 may determine if the size of the couch is less than or equalto the dimensions. If so, the haptic effect determination module 126 maydetermine a haptic effect comprising a stroking sensation. This maynotify the user that the couch will fit in the living room (or in thespace designated for the couch).

Haptic effect generation module 128 represents programming that causesprocessor 102 to generate and transmit haptic signals to the hapticoutput device 118 to generate the selected haptic effect. For example,the haptic effect generation module 128 may access stored waveforms orcommands to send to the haptic output device 118 to create the desiredeffect. In some embodiments, the haptic effect generation module 128 maycomprise algorithms to determine the haptic signal. Further, in someembodiments, haptic effect generation module 128 may comprise algorithmsto determine target coordinates for the haptic effect (e.g., coordinatesfor a location on the computing device 101 or object at which to outputa haptic effect).

Although the modules 124, 125, 126, 128 are depicted in FIG. 1 asprogram components within the memory 104, in some embodiments, themodules 124, 125, 126, 128 may comprise hardware. For example, modules124, 125, 126, 128 may comprise analog to digital converters,processors, microcontrollers, comparators, amplifiers, transistors, andother analog or digital circuitry.

FIG. 2 is another block diagram showing a system for object manipulationwith haptic feedback according to another embodiment. The system 200 maycomprise one or more remote haptic output devices 206. The system 200may also comprise one or more computing devices 202 a-f.

In the embodiment shown in FIG. 2, the system 200 comprises a wearablecomputing device 202 a, a packaging computing device 202 b, a shelfcomputing device 202 c, a store computing device 202 d, a floorcomputing device 202 e, and a carrier computing device 202 f. A wearablecomputing device 202 a may comprise a computing device 202 a associatedwith a wearable device (e.g., a ring) that is configured to be worn bythe user. A packaging computing device 202 b may comprise a computingdevice that is at least partially embedded within an object's packaging(e.g., an object's wrapper or container). A shelf computing device 202 cmay comprise a computing device that is at least partially embeddedwithin or coupled to a shelf (e.g., for holding a product for sale in astore or a bookshelf). A store computing device 202 d may comprise acomputing device associated with a seller, manufacturer, and/ordistributer of an object. A floor computing device 202 e may comprise acomputing device that is at least partially embedded within or coupledto a floor (e.g., within one or more floor tiles of a store). A carriercomputing device 202 f may comprise a computing device that is at leastpartially embedded within or coupled to a carrying device for holding anobject (e.g., a shopping cart, basket, bag, or backpack). Thesecomputing devices 202 a-f are described in greater detail below.

The computing devices 202 a-f and remote haptic output device(s) 206 maybe connected to a network 204. The network 204 may be any suitablenumber or type of networks or links, including, but not limited to, adial-up network, a local area network (LAN), wide area network (WAN),public switched telephone network (PSTN), a cellular network, a WiFinetwork, the Internet, an intranet or any combination of hard-wiredand/or wireless communication links. In some embodiments, the network204 is a single network. In other embodiments, the network 204 maycomprise two or more networks.

The computing devices 202 a-f and remote haptic output device(s) 206 maydirectly communicate with each other and/or may communicate with eachother via the network 204. For example, the shelf computing device 202 cmay communicate wirelessly with store computing device 202 d (e.g.,using Bluetooth). Further, the computing devices 202 a-f and remotehaptic output device(s) 206 may communicate with one or more remoteservers (e.g., cloud servers, webservers, or other servers), databases,and/or computing devices via the network 204.

As described above, the system 200 may comprise a wearable computingdevice 202 a. The wearable computing device 202 a may be associated witha wearable device comprising, for example, a watch, wristband, hat,sleeve, jacket, collar, glasses, glove, ring, articles of clothing,headband, and/or jewelry. Upon a user interacting with an object, thewearable computing device 202 a may output a haptic effect to a bodypart of the user (e.g., the user's wrist, arm, leg, foot, hand, finger,neck, head, or chest). For example, the wearable computing device 202 amay comprise a ring. Upon the user contacting a product with a finger,the wearable computing device 202 a may output a haptic effectconfigured to squeeze a finger of the user. This may alert the user to acharacteristic of the product. For example, the product may comprise afood product and the haptic effect may be configured to alert the userto a negative characteristic, for example, that the food productcontains a high amount of sodium.

As described above, the system 200 may comprise a packaging computingdevice 202 b. For example, the packaging computing device 202 b may beembedded within or coupled to the packaging of an object. The packagingcomputing device 202 b may comprise a sensor (e.g., an accelerometer,pressure sensor, capacitive sensor, resistive sensor, 3D imaging system,or a LED-based tracking system) configured to detect a user interactionwith an object. For example, the sensor may comprise a pressure sensorembedded within a portion of the object's packaging. The pressure sensormay be able to detect contact by a user. Based on signals from thesensor, the packaging computing device 202 b may be configured to outputa haptic effect (e.g., directly to the user's hand or finger as the usercontacts the object).

As described above, the system 200 may comprise a shelf computing device202 c. The shelf computing device 202 c may comprise a sensor (e.g., apressure sensor, accelerometer, position sensor, 3D imaging system, or aLED-based tracking system) configured to detect a user interaction withan object. For example, the sensor may comprise a pressure sensorpositioned under an object on the shelf. The sensor may be configured todetect a user lifting an object off the shelf. Based on signals from thesensor, the shelf computing device 202 c may be configured to output ahaptic effect.

As described above, the system 200 may comprise a store computing device202 d. The store computing device 202 d may be associated with a seller,manufacturer, and/or distributer of an object. For example, the storecomputing device 202 d may be owned or operated by the store in whichthe object is for sale. The store computing device 202 d may comprisedata (e.g., in a database) associated with the object. For example, thestore computing device 202 d may comprise a name, weight, size, price,discount, manufacturer, list of ingredients, and/or other characteristicassociated with the object. One or more other computing devices 202 a-c,202 e-f may communicate with the store computing device 202 d todetermine a characteristic associated with the object. For example, thewearable computing device 202 a may transmit a query associated with theobject to the store computing device 202 d. The store computing device202 d may receive the query and consult a database to determine thequantity of the object available for sale in the store. The storecomputing device 202 d may transmit a signal associated with thequantity to the wearable computing device 202 a.

In some embodiments, the store computing device 202 d may be configuredto detect a user interaction with an object. For example, the storecomputing device 202 d may be in wired or wireless communication with aninteraction sensor. The interaction sensor may comprise, for example, adepth sensor oriented toward an object. Upon the user interacting withan object, the store computing device 202 d may determine informationassociated with the object. Based on the information, the storecomputing device 202 d may cause the remote haptic output device 206and/or another computing device 202 a-c, 202 e-f to output a hapticeffect. For example, the store computing device 202 d may transmit asignal to a wearable computing device 202 a configured to cause thewearable computing device 202 a to output a haptic effect.

As described above, the system 200 may comprise a floor computing device202 e. The floor computing device 202 e may output a haptic effect(e.g., a vibration) upon the user interacting with an object. Forexample, upon a user touching a product in a store, the floor computingdevice 202 e may output a high-magnitude vibration. The user mayperceive the haptic effect via the user's feet or another body part. Thehigh-magnitude vibration may indicate to the user, for example, that theproduct is on sale. In other embodiments, the floor computing device 202e may comprise a computing device in communication with a remote hapticoutput device 206 that is embedded in or coupled to the floor. Upon theuser interacting with an object, the floor computing device 202 e maycause the remote haptic output device 206 to output a haptic effect(e.g., to the user's feet or another body part).

As described above, the system 200 may comprise a carrier computingdevice 202 f. In some embodiments, the carrier computing device 202 foutputs a haptic effect upon a user interacting with an object. Forexample, a user may be holding a shopping basket comprising a carriercomputing device 202 f. Upon a user grasping a bottle of alcohol, thecarrier computing device 202 f may determine whether the user is under21 years old. If so, the carrier computing device 202 f may output astrong vibration. The user may feel the strong vibration through a handas the user holds the shopping basket. This may notify the user that itis illegal for the user to purchase the object. In other embodiments,the carrier computing device 202 f comprises one or more sensorsconfigured to detect if a user has placed an object in the carryingdevice. For example, the carrier computing device 202 f may comprise orbe embedded within a shopping cart. The carrier computing device 202 fmay detect if a user places an object within the shopping cart. In onesuch embodiment, upon the user placing a product into the shopping cart,the carrier computing device 202 f may determine whether the total priceof all the items in the shopping cart exceeds a threshold. If so, thecarrier computing device 202 f may output a strong vibration.

The system 200 may additionally or alternatively comprise one or morecomputing devices positioned in other locations or configurations. Forexample, the system 200 may comprise a computing device positionedwithin the object. As another example, the system 200 may comprise acomputing device associated with a hand-held device or a graspabledevice (e.g., a mobile phone, e-reader, tablet, wand, stylus, or pen).In some embodiments, the computing device may be, for example,positioned in the user's pants pocket (or shirt pocket). Upon the userinteracting with an object, the computing device may output a hapticeffect to the user's thigh (or chest). In other embodiments, the usermay interact with the object using an intermediary object, for example,as described below with respect to FIG. 5B.

In some embodiments, any of the computing devices 202 a-f may cause anyof the other computing device 202 a-f (and/or the remote haptic outputdevice 206) to output a haptic effect. For example, a user may bewearing glasses comprising a wearable computing device 202 a. Thewearable computing device 202 a may detect the user grabbing a productoff a shelf in a store. The wearable computing device 202 a may transmita signal to a packaging computing device 202 b within the product'spackaging. The signal may cause the packaging computing device 202 b tooutput a haptic effect, e.g., to the user's hand. This may allow thewearable computing device 202 a to cause a haptic effect to be output toa body part of the user (e.g., the user's hand) that the wearablecomputing device 202 a may otherwise be unable to output haptic effectsto (e.g., because the glasses may be positioned on the user's head).

As another example, the packaging computing device 202 b may detect theuser picking an object up off of a shelf in a store. The packagingcomputing device 202 b may transmit a signal to the remote haptic outputdevice 206. The signal may be configured to cause the remote hapticoutput device 206 to output a puff of air at the user's chest. This mayallow the packaging computing device 202 b to cause a haptic effect tobe output to a body part of the user (e.g., the user's chest) to whichthe packaging computing device 202 b may otherwise be unable to outputhaptic effects.

FIG. 3 shows an embodiment of a system for object manipulation withhaptic feedback. The system 300 comprises an object 306 positioned on ashelf 308. The object 306 may comprise a toy, computer, mobile device,automotive component, movie, video game, video game console, appliance,television, medical device, mechanical or electrical component, remotecontrol, food, etc. The system 300 may be associated with a store,warehouse, restaurant, medical center, garage, house, office, apartment,or other location.

In the embodiment shown in FIG. 3, an interaction sensor 315 (e.g., a 3Dimaging system) is oriented toward the object 306. The interactionsensor 315 may detect a user interaction (e.g., tap, touch, gesture on,shake, lift, gesture toward, etc.) with the object 306. The user mayinteract with the object 306, for example, to receive information (e.g.,the type, brand, manufacturer, size, color, nutritional value orinformation, function, content, weight, rating, price, expiration date,power consumption, noise level, etc.) associated with the object 306.The interaction sensor 315 may transmit a sensor signal to one or morecomputing devices 304, 310, 312, 318. Based on the user interaction, oneor more of the computing devices 304, 310, 312, 318 may determineinformation (e.g., by querying a server over the Internet) associatedwith the object 306. The one or more computing devices 304, 310, 312,318 may then output a haptic effect associated with the information.

In this example, the system 300 comprises a wearable computing device304 that includes a wristband or watch. The system 300 also comprises apackaging computing device 310, a shelf computing device 312, a floorcomputing device 318, and a remote haptic output device 314. The remotehaptic output device 314 may be in wired or in wireless communicationwith the computing devices 304, 310, 312, 318. In some embodiments, uponthe user interacting with the object 306, one or more computing devices304, 310, 312, 318 may determine information associated with the object306. For example, the information may comprise whether the object 306 ison sale. One or more of the computing devices 304, 310, 312, 318 maycause the remote haptic output device 314 to output a remote hapticeffect associated with the information. For example, the wearablecomputing device 304 may transmit a signal configured to cause theremote haptic output device 314 to output an ultrasonic pressure wavedirected toward the user's hand. The user may perceive the ultrasonicpressure wave as a vibration on the user's hand. The haptic effect mayindicate to the user that the object 306 is on sale. In someembodiments, the wearable computing device 304 may not cause the remotehaptic output device 314 to output a haptic effect if the object 306 isnot on sale. The lack of a haptic effect may indicate to the user thatthe object 306 is not on sale.

FIG. 4 shows another embodiment of a system for object manipulation withhaptic feedback. In this example, the user 402 is wearing a computingdevice 406 comprising a ring. The user 402 is interacting with an object404 by making a gesture in the air. In some embodiments, the gesture maybe oriented toward the object 404 or within a predefined distance (e.g.,2 feet) from the object 404. For example, the user 402 may wave at,point at, reach for, and/or use a finger to draw a symbol in the air infront of the object 404. The computing device 406 may detect the userinteraction and determine the user's gesture. Based on thecharacteristics of the gesture and/or the object 404, the computingdevice may determine information associated with the object 404. Thecomputing device 406 may then output a haptic effect associated with theinformation.

For example, the user 402 may make a gesture in the air to determineinformation about the object 404, e.g., to determine power consumptioninformation associated with the object 404. In the embodiment shown inFIG. 4, the user is drawing a “P” shape in the air in front of an object404. The computing device 406 may detect a gesture and determine thatthe user is making a “P” shaped gesture. The computing device mayfurther determine that the object 404 comprises a particular electronicdevice (e.g., a particular model of laptop computer). Based on the “P”gesture, the computing device may determine (e.g., via the Internet) theaverage amount of power consumed by the electronic device while inoperation. In some embodiments, if the amount of power is below athreshold (e.g., input by the user or that is an industry standard), thecomputing device 406 may output a haptic effect (e.g., a shortvibration). This may indicate to the user that the object 404 consumesan average amount of power that is acceptable to the user. In otherembodiments, if the amount of power is above a threshold, the computingdevice 406 may output a haptic effect configured to indicate to the user402 that the object 404 consumes an unacceptable amount of power.

FIG. 5A shows still another embodiment of a system for objectmanipulation with haptic feedback. In this example, the user 502 iswearing a watch comprising the computing device 506. The user 502 isinteracting with an object 504 (e.g., a screw driver) by contacting(e.g., tapping or touching) the object 504. The computing device 506 maydetect the contact via interaction sensor 510 (e.g., embedded within thecomputing device 506). The interaction sensor 510 may comprise, forexample, a camera oriented toward the object 504. Based on thecharacteristics of the contact and/or the object 504, the computingdevice 506 may determine information associated with the object 404. Thecomputing device 506 may then output a haptic effect associated with theinformation via a haptic output device 512.

In some embodiments, the user may use an intermediary object (e.g., astylus, pen, cane, or wand) for an interaction, and the computing device506 may detect such an interaction. For example, as shown in FIG. 5B,the user 502 may contact the object 504 with an intermediary object 514comprising a mobile phone. The computing device 506 may detect thecontact and output an associated haptic effect. In some embodiments, theintermediary object 514 comprises the computing device 506 (e.g., ratherthan the computing device 506 being separate from the intermediaryobject 514). For example, rather than the user 502 wearing the computingdevice 506 on a wrist, the mobile phone may be the computing device.

In some embodiments, the user 502 may contact the object 504 todetermine information about the object, e.g., if the object 504 is anacceptable price. The computing device 506 may detect the contact anddetermine the price of the object 504. In some embodiments, if the priceis below a threshold (e.g., input by the user), the computing device 506may output a haptic effect (e.g., a pulsed vibration). This may indicateto the user 502 that the object 504 has a price that is acceptable tothe user 502. In other embodiments, the computing device 506 may performa price comparison with other local stores. The computing device 506 mayoutput a haptic effect (e.g., comprising two pulsed vibrations)configured to notify the user 502 that the price may be cheaper atanother local store (e.g., a store within a 15 mile radius) or throughan online retailer.

In some embodiments, the object 504 comprises a tool (e.g., a hammer,screwdriver, drill, nail, saw, screw, or bolt). A user 502 may bebuilding (or repairing) a piece of furniture. The user 502 may bereading instructions for building the piece of furniture on the display508 of the computing device 506 and, at a particular step, grab thetool. The computing device 506 may detect the contact (e.g., via theinteraction sensor 510) and determine the type of the tool (e.g., a size1 Phillips head screwdriver). The computing device 506 may determinewhether the tool is correct for the project or for the particular stepof the project. If not, the computing device 506 may output an intensevibration, e.g., to the wrist of the user 502 via the haptic outputdevice 512. This may alert the user 502 that the user 502 has picked upthe wrong tool.

As another example, in some embodiments, the object 504 comprises amedical tool. The user 502 may be performing a medical procedure (e.g.,a surgery) and grab the medical tool. The computing device 506 maydetect the contact and determine the type of the medical tool (e.g., ascalpel with a number 11 blade). The computing device 506 may determineif the medical tool is correct for the medical procedure or a particularthe step of the medical procedure. If the medical tool is not correctfor the project or the particular step of the project, the computingdevice 506 may output an intense vibration. This may alert the user 502that the user 502 has picked up the wrong tool, which may prevent injuryto the patient.

The computing device 506 may detect a contact anywhere on the object504, or a contact with a specific location (e.g., the label) on theobject 504. The computing device 506 may detect a location of thecontact. For example, the computing device 506 may detect which portionof the object 504 (e.g., the top, bottom, left side, right side, front,back, a label, an image, a logo, a piece of text, etc.) was contacted bythe user. In some embodiments, the computing device 506 may output ahaptic effect upon the user 502 interacting with a specific portion ofthe object 504. For example, the user 502 may contact a manufacturer'slogo on the object 504. The computing device 506 may detect theinteraction with the manufacturer's logo and output a haptic effectassociated with the manufacturer or brand of the object 504. In someembodiments, the computing device 506 may not output a haptic effect ormay output a different haptic effect if the user 502 interacts withanother portion of the object 504.

In some embodiments, the computing device 506 may detect multiple userinteractions (or a single user interaction comprising multiple contacts)with the object 504. For example, in some embodiments, the object 504may comprise a map. The user 502 may touch the map at a starting pointon the map with a finger. The user 502 may sequentially orsimultaneously (with another finger) touch the map at a destinationlocation. The computing device 506 may detect the user interactions andexecute a mapping application. The computing device 506 may furtherdetermine the distance between the start point and destination location.If the distance is below a threshold, the computing device 506 mayoutput a haptic effect.

In some embodiments, the object 504 may comprise a virtual object. Thevirtual object may be, for example, output on the display 508. In onesuch embodiment, the virtual object comprises images representing fooditems in the user's fridge. For instance, the virtual object maycomprise an image of a bottle of milk, carrots, a piece of meat, etc.The computing device 506 may be configured to detect a user interactionwith the virtual object. For example, a user may double tap on thedisplay (e.g., which may be a touchscreen display) on a locationassociated with the bottle of milk. Based on the user interaction, thecomputing device 506 may determine information associated with thevirtual object. For example, based on the double tap, the computingdevice 506 may determine, for example, how many bottles of milk the userhas in the user's refrigerator. The computing device 506 may output ahaptic effect associated with the information. For example, thecomputing device 506 may output an amount of pulsed vibrations equal tothe number of bottles of milk in the user's refrigerator.

In some embodiments, the computing device 506 outputs a sound (e.g., viaa speaker) based on the user interaction. For example, the computingdevice 506 may detect a user interaction with an object 504 anddetermine the price of the object 504. If the price is below a threshold(e.g., input by the user), the computing device 506 may output the soundof a cash register opening (e.g., “ching ching”). This may notify theuser that the object 504 has an acceptable price. As another example,the computing device 506 may output a beep if the object 504 (e.g., aBlu-Ray DVD) is compatible with another object (e.g., a DVD player)owned by the user 502. This may provide additional information about theobject 504 to the user 502.

In some embodiments, the computing device 506 outputs data on a display508 based on the user interaction. For example, a user may tap on anobject 504 in a store. The computing device 506 may determine the priceof the object 504 in the store and the average price of the object 504(e.g., as determined based on prices from a plurality of stores). If theprice in the store is lower than the average price of the object 504,the computing device 506 may output a single dollar sign on the display508. If the price is higher than the average price of the object 504,the computing device 506 may output three dollar signs on the display508. As another example, the computing device 506 may determine thequantity of the object 504 the store has left in stock (e.g., byquerying a server associated with the store). The computing device 506may output the quantity on a display 508. This may provide additionalinformation about the object 504 to the user 502.

In some embodiments, the computing device 506 outputs a haptic effect,sound, and/or data on a display 508 substantially simultaneously withthe user interaction. For example, the computing device 506 may performa number of operations (e.g., determine that a user interaction with theobject 504 occurred, determine a characteristic of the object 504, anddetermine a haptic effect) and output a haptic effect quickly enoughthat the user perceives the haptic effect as being substantiallysimultaneous with the user interaction.

FIG. 6 shows another embodiment of a system for object manipulation withhaptic feedback. In this example, the user 602 is wearing a computingdevice 606 comprising a wrist band. The user may gesture on a surface ofthe object 604 to interact with the object. For example, the user mayperform a two finger pinch on, move multiple fingers along, or make acheckmark on a surface of the object 604.

In some embodiments, the computing device 406 detects a gesture and,based on the gesture, determines a particular characteristic of theobject 404. For example, in the embodiment shown in FIG. 6, the user 602is interacting with the object 604 by moving a finger longitudinallyupward along a front surface of the object 604. This user may be makingthis gesture to, for example, determine if the height of the object 604is less than a predefined height. Based on this specific gesture, thecomputing device 606 may communicate with a local server over a networkto determine the dimensions of the object 604. In some embodiments, ifthe height of the object 604 is below a threshold, the computing device606 may output a haptic effect (e.g., a low frequency vibration). Thismay indicate to the user that the object 604 is an acceptable height,e.g., can fit into a space in the user's home (such as a cupboard). Inother embodiments, if the height of the object 604 is above thethreshold, the computing device 606 may output another haptic effect,e.g., configured to indicate to the user 602 that the object 604 is notan acceptable height.

FIG. 7 shows still another embodiment of a system for objectmanipulation with haptic feedback. In some embodiments, the user 702 mayinteract with an object 704 by making a gesture using the object 704.For example, the user 702 may move the object 704 in real space (e.g.,using the object to draw a letter or number in the air, rotating theobject, tilting the object, etc.).

In the embodiment shown in FIG. 7, the user 702 is shaking the object704 up and down. The user 702 may shake the object 704 to determine,e.g., whether the object 704 is non-refundable under a return policy.The computing device 706 (e.g., positioned in the object 704) may detectthe shake and determine a return policy associated with the object 704(e.g., by communicating with a store computing device or via theInternet). In some embodiments, the computing device 706 may output ahaptic effect associated with the return policy. For example, if theobject 704 is non-refundable, the computing device 706 may output ahaptic effect comprising three pulsed vibrations. If the object 704 isrefundable, the computing device 706 may output a haptic effectcomprising stroking sensation.

In some embodiments, the user interaction may comprise moving the object704 close to a portion of the user's body (e.g., foot, arm, leg,shoulder, hand, neck, head, back, chest, stomach, thigh, etc.) orcontacting a portion of the user's body with the object 704. Forexample, the user 702 may want to purchase a bottle of soda. The user702 may pick up a bottle of soda and tap his right thigh (e.g., over hiswallet in his right pocket) with the bottle of soda. The computingdevice 706 may detect the tap and determine the quantity of the object704 that the user 702 already has in a refrigerator at home. In someembodiments, the user 702 may have input the quantity into the computingdevice 706. For example, the computing device 706 may execute a shoppinglist application. The user may have input the quantity into the shoppinglist application. The computing device 706 may determine the quantityfrom the shopping list application. In other embodiments, the user 702may have a smart refrigerator or other device capable of determining thecontents of the refrigerator and transmitting the quantity to thecomputing device 706. If the user 702 has a quantity of the productbelow a threshold (e.g., the user 702 has less than three bottles ofsoda), the computing device 706 may not output a haptic effect. If theuser 702 has a quantity of the product above a threshold, the computingdevice 706 may output a buzzing sensation. This may help the user 702make smarter purchasing decisions.

In some embodiments, the computing device 706 may detect a plurality ofuser interactions with an object 704 (e.g., making a gesture in front ofthe object 704, contacting the object 704, making a gesturing along asurface of the object 704, and making a gesture using the object 704).The computing device 706 may determine one or more characteristicsassociated with the object 704 for each detected user interaction. Thecomputing device 706 may further output one or more haptic effectsassociated with one or more of the characteristics. For example, thecomputing device 706 may detect user pointing with a finger toward theobject 704 and output a haptic effect associated with the price of theobject 704. The computing device 706 may also detect the user tapping onthe object 704 and output a haptic effect associated with the make ormodel of the object 704. The computing device 706 may further detect theuser lifting and rotating the object 704 by 90 degrees and output ahaptic effect associated with the star rating of the object 704.

In some embodiments, a user interaction may comprise causing aninteraction between multiple objects. For example, the user 702 may tapthe object 704 against another object. For instance, the user 702 maytap a mobile phone against a mobile phone docking station. In someembodiments, the computing device 706 may detect the interaction betweenthe objects and determine information associated with one or more of theobjects. For example, the computing device 706 may detect theinteraction between the mobile phone and the docking station anddetermine if the docking station is compatible with the mobile phone. Insome embodiments, the computing device 706 may output a haptic effect(e.g., a vibration) associated with the information, e.g., to notify theuser that the two objects are compatible or are not compatible.

As another example, the computing device 706 may detect a user 702interacting with (e.g., tapping, holding, gesturing on, or gesturingtoward) a first object 704 with a first body part and interacting with asecond object 704 with a second body part. For instance, the computingdevice 706 may detect the user 702 interacting with the first object 704with the user's right hand and interacting with the second object 704with the user's left hand. The computing device 706 may determine, forexample, which object 704 is heavier. In some embodiments, if the firstobject 704 is heavier, the computing device 706 may output a hapticeffect (e.g., a vibration) to the user's first body part (e.g., theuser's right hand). If the second object 704 is heavier, the computingdevice 706 may output a haptic effect to the user's second body part(e.g., the user's left hand). In some embodiments, if the objects 704are the same weight (or have weights that are within a predefinedtolerance, such as 0.5 lbs, from one another), the computing device 706may output a haptic effect comprising a pulsed vibration to a pluralityof body parts (e.g., both of the user's hands).

FIG. 8 shows yet another embodiment of a system for object manipulationwith haptic feedback. In this example, the object comprises anotherperson 804. The user 806 may interact with the person 804 by, forexample, shaking hands with the person 804. A computing device 802associated with the user 806 may detect the interaction and determine acharacteristic associated with the person 804. The computing device 802may determine the characteristic, for example, by analyzing images ofthe person 804 from a camera, performing an Internet search (e.g., usingthe person's 804 name), searching a social media website, and/orsearching a database (e.g., a public records database). Other examplesof methods for determining the characteristic are further describedbelow. In some embodiments, the characteristic may comprise, forexample, a name, social security number, net worth, height, age,heritage, hair color, nationality, eye color, medical condition, creditscore, gender, credit card number, username (e.g., for a website oraccount), password, temperament, mood, employer, job, hobby, likes,and/or dislikes. The computing device 808 may output a haptic effectassociated with the characteristic.

In some embodiments, the computing device 802 determines thecharacteristic based on data received (e.g., wirelessly) from anothercomputing device 808 and/or electronic device (e.g., RFID tag). Theother computing device 808 and/or electronic device can be associatedwith the other person 804. For example, the person 804 may be wearing acomputing device 808 comprising a biosensor 810. The biosensor 810 maymeasure a heart rate, temperature, blood pressure, biorhythm, and/orother biological characteristic of the person 804. The computing device808 may transmit this information to the computing device 802 associatedwith the user 806. In some embodiments, the computing device 802 may usethe information as the characteristic. In other embodiments, thecomputing device 802 uses the information to determine a characteristicabout the person 804. For example, the computing device 802 may use theinformation to determine the mood, temperament, or emotional state ofthe person 804. In one such embodiment, the computing device 802 maydetermine that the person 804 is angry based on, e.g., the informationcomprising a high blood pressure. In such an embodiment, the computingdevice 808 may output a haptic effect configured to, for example, warnthe user 806 of danger or that the person 804 is angry.

As another example, the computing device 808 may transmit a usernameassociated with a dating site to the computing device 802. The computingdevice 802 may consult the dating website to determine if the person 804has traits that match one or more traits desired by the user 806. If so,the computing device 802 may output a stroking sensation. This maynotify the user 806 that the person 804 is a potential romantic match.If the person 804 has a trait that is undesirable to the user 806, thecomputing device 802 may output a stinging sensation. This may notifythe user 806 that the person 804 may not be a good romantic match.

As still another example, the person 804 may be a salesmen that isselling a product. The computing device 808 may transmit data associatedwith the product to the computing device 802. The computing device 802may determine if the user 806 would be interested in purchasing theproduct (e.g., by analyzing the purchasing habits of the user 806 orusing data input by the user 806). If so, the computing device 802 mayoutput a high frequency vibration.

In other embodiments, the computing device 802 may analyze one or moreimages and/or physical features associated with the person 804 todetermine the characteristic. For example, the computing device 802 maycapture an image of the person 804 using a camera. The computing device802 may use the image to perform facial recognition, read a name tagassociated with the person 804, and/or otherwise identify the person804. The computing device 802 may then communicate with a server (e.g.,for a social network, dating website, search engine, or personalwebsite) to determine additional characteristics about the person 804.

In some embodiments, the object comprises an animal (e.g., cat, dog,turtle, hamster, ferret, or bird). Upon a user 806 interacting with theanimal, the computing device 802 may determine one or morecharacteristics of the animal (e.g., breed, allergen information,temperament, or mood). The computing device 802 may output a hapticeffect, for example, configured to warn the user 806 that the user 806may be allergic to the animal.

Illustrative Methods for Object Manipulation with Haptic Feedback

FIG. 9 is a flow chart of steps for performing a method for providingobject manipulation with haptic feedback according to one embodiment. Insome embodiments, the steps in FIG. 9 may be implemented in program codethat is executed by a processor, for example, the processor in a generalpurpose computer, a mobile device, or a server. In some embodiments,these steps may be implemented by a group of processors. In someembodiments one or more steps shown in FIG. 9 may be omitted orperformed in a different order. Similarly, in some embodiments,additional steps not shown in FIG. 9 may also be performed. The stepsbelow are described with reference to components described above withregard to computing device 101 shown in FIG. 1.

The method 900 begins at step 901 when the processor 102 receives asensor signal. The processor 102 may receive the sensor signal from theinteraction sensor 132 and/or the additional sensors 130. The sensorsignal may comprise data that is associated with a user interaction withan object. For example, in one embodiment, the data may comprise one ormore images of a store shelf on which an object is sitting. In such anembodiment, the image may comprise features such as the object, theshelf, other nearby objects, and a body part of the user (e.g., a user'sfinger). The user's body part may be contacting or near the object.

The method 900 continues at step 902 when the processor 102 detects auser interaction based on the sensor signal. For example, the computingdevice 101 may be positioned within the object and the interactionsensor 132 may comprise an accelerometer. Upon the user shaking theobject, the processor 102 may detect the shake via sensor signals fromthe accelerometer. As another example, the interaction sensor 132 maycomprise a 3D imaging system oriented toward the object. Upon the usergesturing in front of the object, or along the surface of the object,the processor 102 may analyze images from the 3D imaging system. Uponanalyzing the images, the processor 102 may determine that the pixelscorrelating to the user's hand are oriented in a particular mannerassociated with the user contacting or gesturing near the object.

In some embodiments, the user interaction comprises removing the objectfrom a shelf, table, or other location on which the object ispositioned. For example, the computing device 101 may be embedded in ashelf on which the object sits. An interaction sensor 132 comprising aswitch or a pressure sensor may be positioned on the shelf and below theobject. If the user removes the object from the shelf, the interactionsensor 132 may detect a drop in pressure or a change in the state of theswitch and transmit a sensor signal to the processor 102. Based on thesensor signal, the processor 102 may determine that the user lifted theobject from the shelf. In some embodiments, if the low pressure persistsor the switch remains in a particular state for a predetermined periodof time (e.g., 2 seconds), the processor 102 may determine that the useris holding the object.

In some embodiments, the user interaction comprises positioning thecomputing device 101 within a predefined distance from the object. Forexample, the object may transmit a wireless signal (e.g., an IEEE802.11, Bluetooth, a NFC signal, or RFID signal) to the processor 102(via the network interface device 110). Based on the presence or thestrength of the wireless signal (e.g., if the strength exceeds athreshold), the processor 102 may determine whether the object is withinthe predefined distance (e.g., 3 inches) from the computing device 101.If so, the processor 102 may determine that a user interaction hasoccurred.

As another example, the processor 102 may receive a first set of GPSdata from the interaction sensor 132. The object may also comprise GPSfunctionality and transmit a second set of GPS data to the computingdevice 101. The processor 102 may compare the first set of GPS data tothe second set of GPS data and determine the relative distance betweenthe computing device 101 and the object. If the computing device 101 iswithin a predetermined distance from the object, the processor 102 maydetermine that a user interaction has occurred.

The method 900 continues at step 904 when the processor 102 determines acharacteristic associated with the user interaction. In someembodiments, the processor 102 may analyze signals from the sensor 130and/or the interaction sensor 132 to determine a direction, orientation,pattern, pressure amount, speed, and/or other characteristic of the userinteraction.

For example, in some embodiments, the computing device 101 is positionedwithin the object and the interaction sensor 132 may comprise agyroscope. The processor 102 may determine that the user rotated theobject (and by how much) based on orientation sensor signals from thegyroscope. As another example, in some embodiments, the interactionsensor 132 comprises a 3D imaging system oriented toward the object. Theprocessor 102 may analyze a plurality of images from the 3D imagingsystem to determine the characteristics of a gesture. For instance, theprocessor 102 may analyze a plurality of images and determine that theuser made an “X” shape in the air with a finger. As still anotherexample, in some embodiments, the computing device 101 is positionedwithin the object and the sensor 130 may comprise an accelerometer. Theprocessor 102 may receive one or more sensor signals from theaccelerometer and determine, for example, that the user is shaking theobject.

The method 900 continues at step 906 when the processor 102 determinesan object associated with the user interaction. In some embodiments, theobject may comprise a product in a store (e.g., a box of cereal), a tool(e.g., a hammer or screw driver), a medical device (e.g., a trocar,needle, or heartbeat monitor), an automobile, a human, an animal, etc.

In some embodiments, the processor 102 may associate a user interactionwith a particular object based on the proximity of the computing device101 to the object. Examples include using GPS data and the strength of awireless signal (e.g., as described in step 902 above) to determinewhether the computing device 101 is within a predefined distance to theobject. If so, the processor 102 may associate the object with the userinteraction.

In some embodiments, the processor 102 associates a user interactionwith a particular object based on the proximity of the user interactionto the object. For example, a user may gesture with a finger in front ofan object. The interaction sensor 132 may detect the gesture andtransmit a sensor signal to the processor 102. The processor 102 mayanalyze the sensor signal to determine the distance between the fingerand the object. If the finger is within a particular distance (e.g., 6inches) from the object, the processor 102 may associate the gesturewith the object.

In some embodiments, the processor 102 associates a user interactionwith a particular object based on a direction or orientation of the userinteraction. For example, a user may point at an object the user isinterested in purchasing. The interaction sensor 132 may detect thegesture and transmit a sensor signal to the processor 102. The processor102 may analyze the sensor signal and determine that the point gestureis directed toward the object. Based on the direction of the gesture,the processor 102 may associate the object with the point interaction.

In some embodiments, the processor 102 associates a user interactionwith a particular object by comparing sensor data from the object tosensor data from the computing device 101. For example, a user may weara wristband comprising the computing device 101. The user may pick upand shake an object. The processor 102 may receive a first set ofaccelerometer data associated with the shake from an interaction sensor132 within the wristband. The processor 102 may also receive a secondset of accelerometer data from an accelerometer within the object (e.g.,the object may wirelessly communicate the accelerometer data to thecomputing device 101). The processor 102 may compare the two sets ofaccelerometer data. For example, the processor 102 may compare the timestamps of peaks in the two sets of accelerometer data. If the timedifference between the two time stamps is below a threshold (e.g., 0.5s), the processor 102 may determine that the shake was with respect tothat particular object.

As another example, a user may pick up an object off a shelf The shelfmay comprise a computing device 101 configured detect the object'sremoval and transmit (e.g., wirelessly) a time stamp associated withwhen the object was removed from the shelf. The processor 102 mayreceive the time stamp and compare the time stamp with a time stampassociated with data (e.g., an image) from interaction sensor 132. Ifthe time difference between the two time stamps is below a threshold,the processor 102 may determine that the user picked up that particularobject.

The method 900 continues at step 908 when the processor 102 determines acharacteristic associated with the object. In some embodiments, theprocessor 102 may rely on programming in the object characteristicdetermination module 125 to determine the characteristic. Thecharacteristic may comprise information about the object. For example,depending on the type of object, the characteristic may comprise anutritional fact (if the object is a food product), size, weight, depth,color, texture, shape, dimension, price, discount, product rating (e.g.,a grade or another comparative metric, such as a “star rating”),expiration date, function or utility (e.g., what function the objectperforms or may be used for), available quantity or whether the objectis in stock, restriction on use, warranty, brand, manufacturer,producer, place of production, amount of power consumption or otherelectrical characteristic, an amount of noise, a mechanicalcharacteristic, top speed, average speed, material (e.g., whether theobject is made of wood, glass, ceramic, plastic, and/or metal), and/orcompatibility information (e.g., devices with which the object iscompatible) associated with the object.

In some embodiments, the processor 102 determines a characteristicassociated with the object based on the characteristic associated withthe user interaction. For example, the processor 102 may determinedifferent characteristics associated with an object in response todifferent user interactions. For example, the processor 102 maydetermine the number of calories in a food product upon the user tappingthe food product. The processor 102 may determine the amount of sugar inthe food product upon the user moving a finger along the surface of thefood product in an “S” shape.

In some embodiments, processor 102 determines a characteristicassociated with the object based on a wired or wireless signal (e.g.,from another computing device 101). For example, the object may comprisea computing device 101 or the shelf may comprise a computing device 101.The object or the shelf may transmit information about the object whichcan be received by the processor 102. The information may include, forexample, an identifier (e.g., a name, QR code, bar code, QR code, RFIDcode, or unique identifier), classification (e.g., the type of product,manufacturer, producer, or brand), price, discount, coupon, or function(e.g., what the product does or may be used for) associated with theobject. For example, if the object is a shampoo bottle, the informationmay comprise exactly which bottle of shampoo it is (e.g., number 3 onthe shelf), what kind of product it is (e.g., shampoo), what specificproduct it is (e.g., Hair Company's shampoo for fine hair, or abarcode), or what brand of product it is (e.g., Hair Company brand).

In some embodiments, the processor 102 determines the characteristicusing the Internet or a LAN. The processor 102 may communicate with oneor more servers, databases, and/or webpages via the Internet or a LAN todetermine the characteristic. For example, the processor 102 may querymultiple databases over the Internet (e.g., using the object's bar code)to determine the number of stores proximate to the user (e.g., within 10miles) that sell the object. The processor 102 may also determine whichstore has the cheapest price.

The method 900 continues at step 910 when the processor 102 determineswhether the characteristic matches a criterion. The criterion maycomprise, for example, a nutritional characteristic (e.g., a sodiumlevel, calorie level, nutritional value, whether the object includes acertain food or chemical), an expiration date, a production ormanufacturing characteristic (e.g., if the object was produced locallyor was produced in a specific country), a return policy (e.g., if saleof the object is a final, non-refundable sale), an amount of powerconsumption, an amount of noise (e.g., generated by the object),compatibility information (e.g., whether the object is compatible withanother device), a dimension, a shape, a size, an amount of the objectthe user already owns, and/or any other criterion associated with theobject.

In some embodiments, the user inputs the criterion into the computingdevice 101 (e.g., via a touch screen display, a button, a switch, amouse, a keyboard, or another device in communication with the computingdevice 101). For example, the user may select among a plurality ofavailable criteria, or input criteria, using a mobile application. Themobile application can be executing on a mobile device in communicationwith (or comprising) the computing device 101. The mobile device maytransmit the criteria to the computing device 101, which can store thecriteria in memory 104.

In some embodiments, the criterion comprises an item on a list. Forexample, the user may download a wedding registry list onto thecomputing device 101. As the user browses through stores, the user mayinteract with an appliance. Based on the interaction, the processor 102may determine the name and/or manufacturer of the appliance. If the nameand/or manufacturer of the appliance correspond to an item on thewedding registry list, the processor 102 may determine a match.

In some embodiments, the criterion comprises compatibility information.For example, the user may input the name of a game console (e.g., ownedby the user) into the computing device 101. As the user browses a store,the user may interact with a video game. Based on the interaction, theprocessor 102 may determine the name, bar code, and/or QR code of thevideo game. The processor 102 may determine (e.g., via the Internet or astore computing device) if the video game is compatible with the gameconsole. If the video game is not compatible with the game console, theprocessor 102 may determine that there is no match. If the video game iscompatible with the game console, the processor 102 may determine thatthere is a match.

In some embodiments, a third party (e.g., a doctor, nutritionist,friend, and/or family member) inputs the criterion. For example, theuser may be on a diet. A nutritionist may input dietary information intothe computing device 101. The user may interact with one or more fooditems as the user browses a grocery store. The processor 102 may detectthe interaction and determine if the food item conforms to the diet. Ifthe food item does not conform to the diet, the processor 102 maydetermine that there is not a match. If the food item does conform tothe diet, the processor 102 may determine that there is a match.

In some embodiments, an application (e.g., executing on the computingdevice 101) generates the criterion. For example, the processor 102 mayexecute a financial budgeting application. The financial budgetingapplication may generate a monthly spending limit, e.g., to help theuser budget money. Upon the user interacting with an object (e.g.,placing the object in a shopping cart), the processor 102 may determineif the price of the object exceeds the monthly spending limit. If so,the processor 102 may determine that there is not a match. If the priceof the product does not exceed the monthly spending limit, the processor102 may determine there is a match.

As another example, the processor 102 may execute an analysisapplication. The application may analyze characteristics of the user.For example, the application may comprise a shopping analysisapplication that analyzes the shopping history of the user and/orpurchasing decisions made by the user. Based on the analysis, theapplication may develop criteria against which characteristics of anobject can be compared. For example, upon the user interacting with atie, the processor 102 may determine whether the tie matches (e.g., incolor, style, and/or material) one or more shirts recently purchased bythe user. If so, the processor 102 may determine that there is a match.This may allow a user to quickly sift through a large quantity of ties(or other clothing or products) to find matches with previouslypurchased products. In some embodiments, the processor 102 determines ifthe user already owns the object. If so, the processor 102 may determinea match. If not, the processor 102 may determine that there is not amatch.

In some embodiments, the processor 102 determines a match if acharacteristic of an object is within a predetermined tolerance of acriterion. For example, the processor 102 may determine a match if theprice of an object is within 5% of a predetermined price (e.g., $100).As another example, the processor 102 may determine a match if the colorof an object is within two shades of a dark green color. As stillanother example, the processor 102 may determine a match if the size ofa shoe is within half a shoe size of a men's shoe size 10.

In some embodiments, the processor 102 only determines a match if theprocessor 102 has not previously determined a match associated with theobject. For example, a user may touch a car. The car may be on theuser's wish list on the computing device 101, and thus the processor 102may determine a match. Thereafter, if the user touches the car again,the processor 102 may determine that a match has already occurred andnot determine a match again.

In some embodiments, the processor 102 only determines a match if theprocessor 102 has not previously determined a match associated with theobject within a time period. For example, using the car example above,if the user touches the car again the same day, the processor 102 maydetermine that a match has already occurred. Thus, the processor 102 maynot determine a match again. However, if the user touches the car thenext day, the processor 102 may determine a match.

In some embodiments, the processor 102 only determines a match if theprocessor 102 has not previously determined a match associated with aparticular characteristic of the object. For example, a user may touch asnowboard that costs $250. Because the user may have input criteriacomprising the price of the snowboard being below $300, the processor102 may determine a match. If the user interacts with the snowboardagain, the processor 102 may determine that a match with the pricecriteria has already occurred and not determine a match again.Thereafter, the user may input additional criteria comprising the coretype of the snowboard (e.g., birch). Because this is a differentcriterion, if the user interacts with the same snowboard, and thesnowboard has a birch core, the processor 102 may determine a match withthe birch core criteria.

In some embodiments, the processor 102 only determines a match with aspecific criterion if the processor 102 has not previously determined amatch with the specific criterion within a time period. For example,using snowboard example above, if the user touches the snowboard againthe same day, the processor 102 may determine that a match has alreadyoccurred with the price and core criteria. Thus the processor 102 maynot determine another match. However, if the user touches the snowboardthe next day, the processor 102 may determine a match with one or bothcriteria.

In some embodiments, the processor 102 determines a match if the targetmarket for the product matches a characteristic of the user (e.g., thecriterion may be a characteristic of the user). For example, the usermay input (or the computing device 101 may otherwise determine usingsocial media, the Internet, and/or other resources) an age, height,gender, hair color, mood, and/or other physical or emotionalcharacteristic. If a product is designed for a user with thatcharacteristic (e.g., if the product is designed for men and the user isa male), the processor 102 may determine a match.

The method 900 continues at step 912 when the processor 102 determines ahaptic effect. In some embodiments, the processor 102 determines ahaptic effect if there is a match and/or a haptic effect if there is nota match. For example, the processor 102 may determine a haptic effectcomprising a vibration if there is a match. The processor 102 maydetermine no haptic effect if there is not a match.

In some embodiments, the processor 102 determines a haptic effect basedon a characteristic associated with the object. For example, theprocessor 102 may determine the haptic effect based on the brand,height, width, shape, function, color, material, manufacturer, location,price, quality, and/or expiration date of the object. For instance, asdescribed above, the processor 102 may determine a haptic effectassociated with the object's brand.

In some embodiments, the processor 102 determines a haptic effect thatis configured notify a user of a positive or a negative characteristicof an object. For example, if the object is a food product containing apesticide or chemical, the processor 102 may determine a haptic effectcomprising a stinging sensation. This may alert the user that the foodproduct contains harmful ingredients. As another example, if the objectis a food product containing organic ingredients, the processor 102 maydetermine a haptic effect comprising a stroking sensation. By perceivingeither a stinging sensation or a stroking sensation, the user may benotified of the contents of the food product and guided towardpurchasing or not purchasing the product.

In some embodiments, the processor 102 determines a haptic effect with amagnitude that may be indicative of how “good” or “bad” a purchase maybe. A purchase may be “good,” for example, if the object containshealthy ingredients, has a low price, aligns with a user's spending orpurchasing habits, is targeted toward people with the user'scharacteristics, etc. A purchase may be “bad” if the object containsunhealthy or harmful ingredients, has a high price, does not align withthe user's spending or purchasing habits, is targeted toward people withcharacteristics different from those of the user, etc. For example, theprocessor 102 may determine a haptic effect comprising a low-magnitudevibration if the user interacts with a food product comprising aningredient that the user does not like. The processor 102 may determinea haptic effect comprising a medium-magnitude vibration if the userinteracts with a food product comprising an ingredient that the user ismildly allergic to. The processor 102 may determine a haptic effectcomprising a high-magnitude vibration if the user interacts with a foodproduct comprising an ingredient that the user is highly allergic to.

In some embodiments, the processor 102 may determine a haptic effectbased on multiple criteria. For example, the processor 102 may determineif purchasing an object would align with the user's spending habits,e.g., based on the user's purchasing history. If the processor 102determines that purchasing the object would not align with the user'sspending habits, the processor 102 may determine a haptic effectcomprising, e.g., a vibration. In some embodiments, the processor 102may also determine how “confident” the processor 102 is (e.g., with whatlevel of statistical accuracy the determination was made) thatpurchasing the object does not align with the user's spending habits. Inone such embodiment, the processor 102 may determine the magnitude ofthe haptic effect based on the confidence level. For example, if theconfidence level is high (e.g., greater than 80%), the processor 102 maydetermine that the vibration should have a large magnitude.

As another example, the processor 102 may determine the star rating ofan object. In some embodiments, if the star rating is below three stars,the processor 102 may determine a haptic effect comprising a vibration.If the start rating is three or more stars, the processor 102 maydetermine a haptic effect comprising a stroking sensation. In someembodiments, the processor 102 may further determine if the object iswithin the user's budget. In one such embodiment, the processor 102 maydetermine the frequency of the haptic effect based on whether the objectis within the user's budget. For example, if the object is within theuser's budget, the processor 102 may determine that the haptic effectshould have a low frequency. If the object is not within the user'sbudget, the processor 102 may determine that the haptic effect shouldhave a high frequency. Thus, the user may be able to determine whetherthe object meets multiple criteria based on the characteristics of thehaptic effect.

In some embodiments, the processor 102 determines a haptic effect basedon the characteristic associated with the user interaction. For example,the processor 102 may determine the haptic effect based the type,location, duration, or other characteristics of the user interaction.For instance, the processor 102 may determine a haptic effect comprisinga vibration if the user contacts a specific portion of a product, suchas the nutritional label. As another example, the processor 102 maydetermine a haptic effect if the user interaction comprises a swipealong a surface of the object, and no haptic effect if the userinteraction comprises tapping on the object.

In some embodiments, the computing device 101 may store associated“haptic profiles” in which a user can determine and save in memory 104 a“profile” of the haptic effects the user would like associated withparticular characteristics of an object. For example, in one embodiment,a user can select from a list of options which haptic effect the userwould like associated with the price, battery life, nutritional value,or material of an object. In some embodiments, the list may comprise,for example, haptic effects such as low magnitude vibration, pulsedvibration, high-magnitude vibration, or a simulated texture. In someembodiments, the processor 102 may consult with the user's hapticprofile to determine which haptic effect to generate. For example, ifthe user's haptic profile associates locally produced products with ahaptic effect comprising a low-frequency vibration, in response to theuser contacting a locally made chocolate bar, the processor 102 maydetermine a haptic effect comprising a low-frequency vibration.

In some embodiments, the processor 102 may determine a default hapticeffect. For example, if the processor 102 is unable to perform anoperation (e.g., unable to determine a characteristic associated with auser interaction, a characteristic associated with an object, acriterion, whether a characteristic associated with an object matchesthe criterion, and/or a haptic effect), the processor 102 may determinea haptic effect comprising a vibration. This may alert the user that theoperation could not be performed.

For example, the computing device 101 may be configured to output a joltif the user interacts with a food item containing an allergen. Thecomputing device 101 may be configured to output no haptic effect if theuser interacts with a safe food item. In such an embodiment, if theprocessor 102 cannot determine the ingredients of a food item (e.g.,because it cannot connect to the Internet), the processor 102 maydetermine a haptic effect comprising three pulsed vibrations. Thus,rather than the computing device 101 outputting no haptic effect, whichcould falsely lead the user to believe the food item is safe, the threepulsed vibrations can alert the user that the processor 102 was unableto determine the contents of the food. The user can then read the labelor otherwise determine whether the food item has an allergen.

In some embodiments, the processor 102 determines a haptic effect thatis configured to simulate the feeling of the object while in operation.For example, if the object comprises an electric drill, the hapticeffect may comprise vibrations with magnitudes corresponding to theactual magnitude of vibrations generated by the drill while inoperation. This may allow the user to perceive what operating the drillmay actually feel like while in operation, and thereby make a smarterpurchasing decision.

In some embodiments, the processor 102 determines a haptic effect thatis configured to indicate (or simulate) the function of the object. Forexample, if the object comprises a car, the processor 102 may determinea haptic effect comprising a rumbling vibration, e.g., to simulate acar's engine. If the object comprises a bottle opener, the processor 102may determine a haptic effect comprising a pop sensation, e.g., tosimulate popping the top off a bottle. This may allow the user todetermine the function of the object (e.g., without looking at theobject).

In some embodiments, the processor 102 determines a haptic effect basedon a characteristic of the user. For example, if the processor 102determines that the user is a male, the processor 102 may determine adifferent haptic effect than if the processor 102 determines that theuser is a female. As another example, the processor 102 may determine ahaptic effect if the user is under 40 years old than if the user is over40 years old. In this manner, haptic effects can be customized fordifferent demographics, which may improve a user's response to thehaptic effect.

In some embodiments, the processor 102 determines a haptic effect basedon a position of the computing device 101 in real space or with respectto an object. For example, as the distance between the computing device101 and the object increases, the haptic effect determination module 126may determine a haptic effect comprising a decreasing magnitude. As thedistance between the computing device 101 and the object decreases, thehaptic effect determination module may determine a haptic effectcomprising an increasing magnitude. Thus, the haptic effect may guidethe user toward or away from an object, or indicate to the user how farthe user is from the object.

In some embodiments, the processor 102 determines a plurality of hapticeffects. Each of the plurality of haptic effects may be associated witha different characteristic of the object. For example, the processor 102may determine a haptic effect associated with the size of the object(e.g., a long vibration, which may be configured to indicate that theobject will fit within a pre-designated space). The processor 102 mayalso determine a haptic effect associated with the star rating of theobject (e.g., a number of pulsed vibrations corresponding to the numberof stars the object has). Based on the plurality of haptic effects, theuser may be able to determine multiple characteristics of the object(e.g., that the object will fit in the user's house and has four stars),for example, without having to measure the object or browse forinformation about the object on the Internet (which may be tedious andtime consuming).

In some embodiments, the processor 102 determines that a specific hapticeffect has a higher priority than another haptic effect, and thus tooutput only the high priority effect. For example, in the above size andstar rating example, the processor 102 may determine that the hapticeffect associated with the size has a higher priority than other effects(e.g., the haptic effect associated with the star rating), and thusoutput only the size haptic effect. Alternatively, the processor 102 maydetermine that only the most intense effect should be output. Thus, insome embodiments, the processor 102 may determine a low intensityvibration and a high intensity vibration, but output only the highintensity vibration.

In some embodiments, the processor 102 may determine a haptic effectconfigured to make it easier or harder to interact with the object. Forexample, the object may be positioned on a shelf. Upon a userapproaching the object, the processor 102 may determine that the objectcomprises a drug harmful to the user. Based on the harmful drug, theprocessor 102 may transmit a signal to a packaging computing device. Thepackaging computing device may receive the signal and output a hapticeffect configured to lower the perceived coefficient of friction on thesurface of the object's packaging. This may make it challenging for theuser to grip or grasp the object to pick it up. Further, in someembodiments, the object (e.g., packaging of the object) may have a metalbottom. Based on the harmful drug, the processor 102 may transmit asignal to a shelf computing device. The shelf computing device mayreceive the signal and actuate an electromagnetic device configured toapply a strong magnetic field to the metal bottom of the object. Thismay make it challenging for the user to lift the object off the shelf(e.g., to purchase it).

As another example, the object may be deformed around another object,such as a shelf (e.g., to secure the object to the shelf). For instance,the shelf may comprise a bar around which the object is bent to securethe object to the bar. In some embodiments, upon the user interactingwith the object, the processor 102 may transmit data (e.g., a purchasereceipt code, name, credit card number, or customer identifier) to theobject (e.g., another computing device 101 positioned in the object).The data may indicate that the user already purchased the item (e.g.online), or may otherwise be associated with a purchase of the product(e.g., the data may include a credit card number which the object mayuse to conduct the purchase in real time). Based on the data, the objectmay output a haptic effect configured to deform the shape of the object.For example, the object may output a haptic effect configured to unbendthe object from around the bar (e.g., to de-secure the object from theshelf). This may allow the user to take the object.

In some embodiments, the object may be positioned on a shelf. Amechanical coupling device may be positioned between the object and theshelf. Upon a user approaching the object, the processor 102 maydetermine that the object has a negative characteristic (e.g., that theobject comprises a chemical harmful to the user). The processor 102 maytransmit a signal to the coupling device. Based on the signal, thecoupling device may couple or latch to the object to the shelf, makingit challenging for the user to lift the object.

The method 900 continues at step 914 when the processor 102 outputs thehaptic effect. The processor 102 may transmit a haptic signal associatedwith the haptic effect to haptic output device 118, which outputs thehaptic effect. The haptic effect may comprise a texture (e.g., sandy,bumpy, or smooth), a vibration, a change in a perceived coefficient offriction, a change in temperature, a stroking sensation, anelectro-tactile effect, or a deformation (i.e., a deformation of asurface associated with the computing device 101).

Additionally or alternatively, in some embodiments, the computing device101 may output a sound and/or information on a display. The sound and/orinformation may be associated with a characteristic of the object. Forexample, the user may tap on a toy. The computing device 101 maydetermine the name of the toy and, based on the name, whether the toy isa present on a holiday shopping list (e.g., input by the user, theuser's family members, and/or the user's friends). If so, the computingdevice 101 may play a portion of a holiday song via a speaker. Thecomputing device 101 may also output the name of the person (e.g., afriend or family member) associated with the toy on a display. Forexample, the computing device 101 may output “Vincent's present” on thedisplay. If the holiday list includes presents for multiple people, orwas made by a third party, this may help the user determine who theparticular present is for.

Advantages of Object Manipulation with Haptic Feedback

There are numerous advantages to object manipulation with hapticfeedback. Such systems may provide information about an object quicklyand efficiently. For example, a user can tap on an object andsubstantially simultaneously receive information about the price orrating of an object (e.g., via haptic effects). This may be quicker andeasier than, for example, looking up such information using a mobiledevice or consulting with a store employee.

In some embodiments, object manipulation with haptic feedback can guidea user toward making a good purchase or away from making a bad purchase.For example, a computing device may provide a user with positive hapticfeedback (e.g., a light vibration or stroking sensation) if a foodproduct conforms with a user's diet or negative haptic feedback (e.g., ashock or strong vibration) if the food product is harmful to the user'shealth. As another example, upon a user tapping on a product the user isinterested in purchasing, the computing device can output a hapticeffect configured to alert a user that the user already owns theproduct.

In some embodiments, object manipulation with haptic feedback can guidea user to a location associated with an object. For example, a nurse maybe wearing a computing device comprising a ring. The nurse may pick up abottle of medication and put it down in a location where it does notbelong. Upon the nurse putting the medication down in the wronglocation, the computing device may output a strong vibration to thenurse's finger. This may alert the nurse that the nurse has put themedication in the wrong location.

In some embodiments, object manipulation with haptic feedback canprovide safety or medical information to a user. For example, apharmacist may be holding a bottle of medication and move to pick upanother bottle of medication. If the two would cause injury if mixedtogether or otherwise do not go together, the computing device mayoutput a pulsed vibration. This may alert the pharmacist that the twomedications should not be mixed or handled together.

In some embodiments, object manipulation with haptic feedback canimprove branding and provide an enhanced marketing experience. Forexample, upon a user interacting with a product or a logo on theproduct, the computing device may output a haptic effect associated withthe brand of the product. This may allow a user to identify themanufacturer of a product while not visually focused on the product.This may also generate a stronger relationship between the user and thebrand.

General Considerations

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and/or various stages may be added, omitted, and/or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail to avoid obscuringthe configurations. This description provides example configurationsonly, and does not limit the scope, applicability, or configurations ofthe claims. Rather, the preceding description of the configurations willprovide those skilled in the art with an enabling description forimplementing described techniques. Various changes may be made in thefunction and arrangement of elements without departing from the spiritor scope of the disclosure.

Also, configurations may be described as a process that is depicted as aflow diagram or block diagram. Although each may describe the operationsas a sequential process, many of the operations can be performed inparallel or concurrently. In addition, the order of the operations maybe rearranged. A process may have additional steps not included in thefigure. Furthermore, examples of the methods may be implemented byhardware, software, firmware, middleware, microcode, hardwaredescription languages, or any combination thereof. When implemented insoftware, firmware, middleware, or microcode, the program code or codesegments to perform the necessary tasks may be stored in anon-transitory computer-readable medium such as a storage medium.Processors may perform the described tasks.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, in which other rules may takeprecedence over or otherwise modify the application of the invention.Also, a number of steps may be undertaken before, during, or after theabove elements are considered. Accordingly, the above description doesnot bound the scope of the claims.

The use of “adapted to” or “configured to” herein is meant as open andinclusive language that does not foreclose devices adapted to orconfigured to perform additional tasks or steps. Additionally, the useof “based on” is meant to be open and inclusive, in that a process,step, calculation, or other action “based on” one or more recitedconditions or values may, in practice, be based on additional conditionsor values beyond those recited. Headings, lists, and numbering includedherein are for ease of explanation only and are not meant to belimiting.

Embodiments in accordance with aspects of the present subject matter canbe implemented in digital electronic circuitry, in computer hardware,firmware, software, or in combinations of the preceding. In oneembodiment, a computer may comprise a processor or processors. Theprocessor comprises or has access to a computer-readable medium, such asa random access memory (RAM) coupled to the processor. The processorexecutes computer-executable program instructions stored in memory, suchas executing one or more computer programs including a sensor samplingroutine, selection routines, and other routines to perform the methodsdescribed above.

Such processors may comprise a microprocessor, a digital signalprocessor (DSP), an application-specific integrated circuit (ASIC),field programmable gate arrays (FPGAs), and state machines. Suchprocessors may further comprise programmable electronic devices such asPLCs, programmable interrupt controllers (PICs), programmable logicdevices (PLDs), programmable read-only memories (PROMs), electronicallyprogrammable read-only memories (EPROMs or EEPROMs), or other similardevices.

Such processors may comprise, or may be in communication with, media,for example tangible computer-readable media, that may storeinstructions that, when executed by the processor, can cause theprocessor to perform the steps described herein as carried out, orassisted, by a processor. Embodiments of computer-readable media maycomprise, but are not limited to, all electronic, optical, magnetic, orother storage devices capable of providing a processor, such as theprocessor in a web server, with computer-readable instructions. Otherexamples of media comprise, but are not limited to, a floppy disk,CD-ROM, magnetic disk, memory chip, ROM, RAM, ASIC, configuredprocessor, all optical media, all magnetic tape or other magnetic media,or any other medium from which a computer processor can read. Also,various other devices may comprise computer-readable media, such as arouter, private or public network, or other transmission device. Theprocessor, and the processing, described may be in one or morestructures, and may be dispersed through one or more structures. Theprocessor may comprise code for carrying out one or more of the methods(or parts of methods) described herein.

While the present subject matter has been described in detail withrespect to specific embodiments thereof, it will be appreciated thatthose skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, it should be understoodthat the present disclosure has been presented for purposes of examplerather than limitation, and does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

What is claimed:
 1. A computing device comprising: a sensor configured to detect a user interaction with a physical object and transmit a sensor signal associated with the user interaction; a processor in communication with the sensor, the processor configured to: receive the sensor signal; determine a characteristic of the physical object based on the sensor signal; determine a haptic effect associated with the characteristic; and transmit a haptic signal associated with the haptic effect; and a haptic output device in communication with the processor, the haptic output device configured to receive the haptic signal and output the haptic effect.
 2. The computing device of claim 1, wherein the processor is further configured to: determine whether the characteristic matches at least one criterion; and determine the haptic effect based on whether the characteristic matches the at least one criterion.
 3. The computing device of claim 2, wherein the at least one criterion is generated by an application or is input by a third party.
 4. The computing device of claim 1, wherein the processor is further configured to: determine a second characteristic associated with the user interaction; and determine the haptic effect based at least in part on the second characteristic.
 5. The computing device of claim 1, wherein the physical object comprises a product for sale in a location.
 6. The computing device of claim 5, wherein the processor is further configured to determine the characteristic by wirelessly communicating over a network with a server associated with the location.
 7. The computing device of claim 1, wherein the user interaction comprises making a gesture in real space or making the gesture using the physical object.
 8. The computing device of claim 1, wherein the computing device is associated with a wearable device comprising a shoe, an armband, a sleeve, a jacket, glasses, a glove, a ring, a watch, a wristband, a bracelet, an article of clothing, a hat, a headband, or jewelry, or wherein the computing device is associated with a graspable device comprising a mobile phone, a tablet, an e-reader, a portable gaming device, a wand, a stylus, or a pen.
 9. The computing device of claim 1, wherein the computing device is coupled to a shelf, a floor, or a carrying device.
 10. A method comprising: receiving a sensor signal from a sensor, wherein the sensor signal is associated with a user interaction with a physical object; determining a characteristic associated with the physical object based on the sensor signal; determining a haptic effect associated with the characteristic; and transmitting a haptic signal associated with the haptic effect to a haptic output device, wherein the haptic output device is configured to receive the haptic signal and output the haptic effect.
 11. The method of claim 10, further comprising: determining whether the characteristic matches at least one criterion; and determining the haptic effect based on whether the characteristic matches the at least one criterion.
 12. The method of claim 11, wherein the at least one criterion is generated by an application or is input by a third party.
 13. The method of claim 10, further comprising: determining a second characteristic associated with the user interaction; and determining the haptic effect based at least in part on the second characteristic.
 14. The method of claim 10, wherein the physical object comprises a product for sale in a location, and further comprising determining the characteristic by communicating over a network with a server associated with the location or the physical object.
 15. The method of claim 10, wherein the sensor is associated with a wearable device comprising a shoe, an armband, a sleeve, a jacket, glasses, a glove, a ring, a watch, a wristband, a bracelet, an article of clothing, a hat, a headband, or jewelry, or wherein the sensor is associated with a graspable device comprising a mobile phone, a tablet, an e-reader, a portable gaming device, a wand, a stylus, or a pen.
 16. The method of claim 10, wherein the sensor is coupled to a shelf, a floor, or a carrying device.
 17. A non-transient computer readable medium comprising program code, which when executed by a processor is configured to cause the processor to: receive a sensor signal from a sensor, wherein the sensor signal is associated with a user interaction with a physical object; determine a characteristic associated with the physical object based on the sensor signal; determine a haptic effect associated with the characteristic; transmit a haptic signal associated with the haptic effect to a haptic output device, wherein the haptic output device is configured to receive the haptic signal and output the haptic effect.
 18. The non-transient computer readable medium of claim 17, further comprising program code, which when executed by the processor is configured to cause the processor to: determine whether the characteristic matches at least one criterion; and determine the haptic effect based on whether the characteristic matches the at least one criterion.
 19. The non-transient computer readable medium of claim 17, further comprising program code, which when executed by the processor is configured to cause the processor to: determine a second characteristic associated with the user interaction; and determine the haptic effect based at least in part on the second characteristic.
 20. The non-transient computer readable medium of claim 17, wherein the sensor is coupled to a shelf, a floor, or a carrying device. 